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This  paper  focuses  on  improvement  of  test  and  evaluation  (T&E),  and  identifies:  the 
high  levels  of  oversight  with  regard  to  T&E,  the  key  players  in  the  acquisition  process 
and  their  required  interactions,  the  purpose  of  testing  and  recent  reforms  in  testing. 
Seven  recommendations  are  provided  to  improve  test  and  evaluation,  including:  allow 
for  greater  access  to  and  use  of  contractor  test  data,  establish  adequate  testing  to 
confirm  requirements,  increase  accountability  among  stakeholders,  consolidate  reviews 
and  eliminate  redundancy,  improve  the  requirements  process,  support  risk  based 
testing,  and  include  the  T&E  community  in  configuration  steering  boards. 


Improving  Army  Test  and  Evaluation:  A  Team  Sport 


Institutional  change  is  not  merely  about  pinching  pennies  or  pushing  pens. 
And  efficiencies  are  not  simply  about  improving  the  bottom  line.  They’re 
about  doing  things  better,  doing  them  smarter,  and  taking  full  advantage  of 
the  progress,  technology,  knowledge,  and  experience  that  we  have 
available  to  us. 


— The  Honorable  John  McHugh1 
Secretary  of  the  Army 


The  defense  acquisition  system  is  complex,  and  it  poses  unique  challenges  to 
the  Department  of  Defense  in  its  effort  to  develop  and  field  new  capabilities.  The  Army 
has  struggled  to  determine  its  modernization  priorities  since  the  cancellation  of  the 
Future  Combat  System  in  2009.  Acquisition  challenges  coupled  with  undefined  priorities 
have  resulted  in  the  cancellation  of  several  Major  Defense  Acquisition  Programs 
(MDAP).  As  a  result,  Department  of  Defense  leaders  are  losing  faith  in  the  acquisition 
system’s  ability  to  successfully  deliver  capabilities.  From  1990  to  2010,  the  Army 
terminated  twenty-two  MDAPs  before  completion,  with  15  having  been  terminated  since 
2001 .  This  represents  approximately  a  quarter  of  the  Army’s  allocated  research, 
development,  test  and  evaluation  money  for  this  time  period  spent  on  failed  programs.2 

As  a  result  of  these  problems,  Army  leadership,  the  Office  of  the  Secretary  of 
Defense  (OSD),  Congress  and  industry  have  lost  trust  in  the  Army’s  acquisition 
processes  and  capability  to  effectively  provide  Warfighters  the  equipment  and  services 
they  require  in  a  timely  manner.  Despite  these  struggles,  the  U.S.  Army  is  still  regarded 
as  the  best  equipped  and  most  technologically  advanced  army  in  the  world.  This  can  be 
attributed  in  part  to  overseas  contingency  operations  funding  and  rapid  acquisition 
processes  employed  during  the  last  ten  years  which  have  delivered  cutting  edge 


technologies  to  the  battlefield.  In  light  of  the  impending  fiscal  uncertainty  and  defense 
budget  reductions,  supplemental  funding  will  soon  come  to  a  halt.  As  a  result,  the 
acquisition  community  must  find  more  timely  and  efficient  means  to  provide  Warfighters 
the  capabilities  needed  to  remain  the  best  equipped  Army  in  the  world.3 

The  current  uncertain  fiscal  environment  facing  the  Nation  forces  the  Army  to 
revisit  its  approach  to  developing,  acquiring  and  sustaining  new  acquisition  systems. 
This  approach  must  result  in  the  delivery  of  effective  and  reliable  systems,  but  it  must 
also  be  flexible  and  adaptive  in  its  application  such  that  timely  decisions  are  made 
based  on  demonstrated  performance  and  reasonable  risk.  This  paper  will  analyze  how 
the  Army’s  test  and  evaluation  efforts  support  the  larger  acquisition  process,  and  it  will 
identify  areas  of  improvement  to  speed  delivery  of  solutions  to  the  Warfighter  more 
efficiently  and  economically  without  compromising  quality  or  accepting  undue  risk. 

Background 

The  purpose  of  Army  acquisition  is  to  equip  and  sustain  the  Army  so  that  it  meets 
current  and  future  mission  requirements.  The  manner  in  which  the  Army  does  this  is 
complex  and  is  guided  by  20  different  statutes,  2,000  pages  of  regulations  and  other 
various  policies.4  The  law  requires  materiel  systems  be  procured  through  a  three  step 
process:  1)  identifying  the  capability  required,  2)  establishing  a  budget  to  support  the 
effort  and  3)  acquiring  the  system  in  accordance  with  the  laws  and  guidelines  of  the 
Defense  Acquisition  System.5  Successful  implementation  and  execution  requires  that 
these  processes  be  aligned. 

Requirements  are  identified  by  the  Joint  Staff,  the  Services,  and  the  Combatant 
Commanders  through  a  process  known  as  the  Joint  Capabilities  Integration  and 
Development  System  (JCIDS).  JCIDS  assesses  gaps  in  military  capabilities  and 
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recommends  solutions  to  fill  the  gaps.  It  plays  a  key  role  in  identifying  the  capabilities 
required  by  the  Warfighters  to  support  the  National  Security  Strategy,  the  National 
Defense  Strategy  and  the  National  Military  Strategy.  The  primary  objective  of  the  JCIDS 
process  is  to  ensure  that  capabilities  required  by  joint  Warfighters  to  successfully 
execute  the  missions  assigned  to  them  are  identified.  The  JCIDS  process  also  identifies 
the  operational  performance  criteria  of  capabilities  required  by  the  Joint  Force.6  The 
requirements  process  supports  the  acquisition  process  by  providing  validated  capability 
needs  and  associated  performance  criteria  to  be  used  as  a  basis  for  acquiring  the  right 
systems.7  For  materiel  solutions,  JCIDS  produces  the  Initial  Capabilities  Document, 
Capability  Development  Document  (CDD),  and  the  Capabilities  Production  Document 
(CPD).  The  CDD  and  CPD  are  system  specific  and  support  development  and  evaluation 
by  documenting  the  technical  requirements  the  system  is  expected  to  meet.  The  Vice 
Chairman,  Joint  Chiefs  of  Staff  is  the  chairman  of  the  Joint  Requirements  Oversight 
Council  (JROC)  that  oversees  the  JCIDS  and  supervises  the  preparation  of  the 
Chairman’s  Program  Recommendation,  which  provides  recommendations  to  OSD  for 
inclusion  in  the  Defense  Planning  and  Programming  Guidance.  The  Vice  Chief  of  Staff 
of  the  Army  is  the  Army  representative  on  the  JROC.8 

The  budget  is  established  through  the  Planning,  Programming,  Budgeting  and 
Execution  System  (PPBE).  The  purpose  of  the  PPBE  process  is  to  allocate  resources 
within  the  Department  of  Defense.  The  process  is  a  combination  of  four  distinct 
sections:  Planning,  Programming,  Budgeting,  and  Execution.  PPBE  is  linked  to  the 
Defense  Acquisition  System  by  the  financial  resources  it  provides  to  acquisition 
programs.  Upon  initiation,  an  acquisition  program  identifies  its  resource  requirements 
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over  the  life  of  the  program.  These  requirements  must  be  consistent  with  the  resources 
that  have  been  allocated  by  the  PPBE  process.  As  the  program  proceeds  through  the 
acquisition  process,  its  budget  requirements  are  updated.  Any  changes  to  budget 
requirements  must  be  addressed  through  the  PPBE  process.  Decisions  that  alter  the 
program’s  budget  request,  whether  through  additional  funding  or  funding  cuts,  have  an 
effect  on  the  program’s  execution.  For  instance,  budget  cuts  could  reduce  the  scope  of 
a  program;  extend  its  schedule  or  both.9 

The  Defense  Acquisition  System  is  the  process  used  to  develop  and  purchase 
the  desired  system.  The  two  guiding  documents  that  outline  this  process  are  DoD 
Directive  5000.01  and  5000.02.  DoD  Directive  5000.01,  The  Defense  Acquisition 
System,  provides  the  policies  and  principles  that  govern  the  Defense  Acquisition 
System.10  DoD  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System, 
establishes  the  management  framework  that  implements  these  policies  and  principles.11 
The  Defense  Acquisition  System  is  an  event-based  process  where  programs  advance 
through  several  decision  points  and  milestone  reviews  until  ultimately  being  fielded  to 
the  Services. 

Primary  Participants  in  Defense  Acquisition 

The  major  participants  in  the  acquisition  process  are  Congress,  the  Office  of 
Management  and  Budget,  the  OSD,  the  Joint  Staff,  the  Office  of  the  Service  Secretary, 
the  Service  Acquisition  Executive,  the  military  service  materiel  commands,  program 
management  offices  (PMO),  industry  and  functional  support  organizations.  Army 
Training  and  Doctrine  Command  (TRADOC)  and  the  Army  Test  and  Evaluation 
Command  (ATEC)  are  the  two  primary  functional  support  organizations  for  Army 


4 


acquisition.  Each  participant  in  the  acquisition  process  exercises  responsibilities  to 
ensure  that  laws  and  regulations  are  observed  and  programs  pursued  efficiently. 

This  paper  focuses  on  improvement  of  test  and  evaluation  by  identifying:  the  high 
levels  of  oversight  with  regard  to  T&E,  the  key  players  in  the  acquisition  process  and 
their  required  interactions,  the  purpose  of  testing  and  recent  reforms  in  testing.  The 
recommendations  at  the  end  of  this  discussion  will  improve  test  and  evaluation  when 
implemented. 

OSD,  through  the  Director,  Operational  Test  and  Evaluation  (DOT&E)  provides 
operational  evaluation  reports  to  Congress.  The  primary  purpose  of  DOT&E  is  to 
describe  the  operational  effectiveness  and  suitability  of  a  system  being  tested  within  the 
operational  combat  environment.12  DOT&E  was  created  in  part  because  of  the  findings 
of  a  Blue  Ribbon  Panel  Report  in  1970  which  cited  problems  with  operational  testing 
and  evaluation.  The  report  noted  that  each  Service  had  a  different  system  for 
operational  testing  that  neither  OSD  nor  the  Joint  Chiefs  had  much  control  over. 
Eventually  established  by  Congress  in  1983,  DOT&E  was  tasked  with  providing  test  and 
evaluation  oversight  in  order  to  help  coordinate  Service  testing  policies  and  to  ensure 
that  the  Services  devoted  adequate  resources  to  this  area.13  The  Weapon  Systems 
Acquisition  Reform  Act  of  2009  established  The  Office  of  the  Deputy  Assistant 
Secretary  of  Defense,  Developmental  Test  and  Evaluation  (DT&E)  within  the  Office  of 
the  Assistant  Secretary  of  Defense,  Research  and  Engineering.  DT&E  is  responsible  for 
evaluating  developmental  test  capability  within  the  Department  of  Defense.  The  creation 
of  this  agency  highlights  the  importance  DoD  places  on  developmental  testing  in  its 
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relation  to  systems  acquisition.  DT&E  is  designed  to  be  a  collaborative  partner  with 
DOT&E  to  ensure  acquisition  decisions  are  supported  with  the  right  T&E  information.14 

The  role  of  the  PMO  is  to  direct  the  development,  production,  and  fielding  of  a 
new  defense  system.  This  must  be  done  within  the  limits  of  cost,  schedule,  and 
performance,  as  approved  by  the  program  manager's  acquisition  executive.  The 
program  manager's  role  is  to  be  the  Army’s  agent  to  ensure  the  Warfighter's 
modernization  requirements  are  met  efficiently  and  effectively  in  the  shortest  possible 
time.15 

TRADOC,  through  TRADOC  Capability  Managers  (TCMs),  represents  the 
Warfighter  by  serving  as  the  advocate  for  the  end-user.  TCMs  are  responsible  for 
systems  that  provide  a  particular  capability  or  function  for  the  Army.  The  TCMs  provide 
organizational  subject  matter  expertise  for  their  assigned  capability  area  and  are 
responsible  for  the  technical  and  functional  requirements  of  the  materiel  systems. 
Furthermore,  TCMs  provide  for  the  integration  of  all  doctrine,  organization,  training, 
materiel,  leadership  and  education,  personnel,  and  facilities  (DOTMLPF)  for  the 
capability  area.16 

ATEC  serves  as  the  independent  evaluator  of  Army  systems.  The  Army 
Evaluation  Center  performs  evaluation  planning  and  uses  the  test  resources,  including 
ATEC’s  developmental  test  ranges  as  well  as  the  Operational  Test  Command,  to  carry 
out  detailed  evaluations  that  encompass  system  Effectiveness,  Suitability,  and 
Survivability  in  an  operational  context.  ATEC's  primary  purpose  is  to  provide 
independent  information  to  decision  makers  to  aid  in  acquisition  decisions.17 
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Although  each  organization  has  different  missions,  they  all  work  together  toward 
a  common  end  -  the  delivery  of  effective,  suitable  and  survivable  systems.  The  following 
sections  of  this  paper  will  focus  on  the  interactions  between  OSD,  PMOs,  TCMs  and 
ATEC  representatives  as  they  work  towards  fielding  new  equipment. 

Introduction  to  Testing 

“The  primary  purpose  of  test  and  evaluation  (T&E)  is  to  support  system 
development  and  acquisition  by  serving  as  a  feedback  mechanism  in  the  iterative 
systems  engineering  process.”18  ATEC’s  mission  is  to  provide  information  to  acquisition 
decision  makers.  Through  testing,  the  T&E  community  determines  if  a  system  meets  its 
specified  requirements  to  ascertain  whether  it  provides  the  military  capability  being 
sought.  A  variety  of  testing  is  planned  and  performed,  from  early  contractor  tests 
through  rigorous  operational  tests,  all  designed  to  evaluate  the  progress  of  the  program. 
It  is  through  testing  that  the  Army  validates  the  performance  against  the  requirements 
identified  during  the  JCIDS  as  outlined  in  the  requirements  documents  (CDD  or  CPD). 
These  documents  outline  the  threshold  and  objective  requirements  of  a  particular 
system  as  well  as  the  Key  Performance  Parameters  (KPPs).  The  threshold  requirement 
denotes  the  minimum  acceptable  requirements  and  the  objective  denotes  the  desired 
requirements.  The  KPP  is  the  capability  or  characteristic  that  is  so  significant  that  failure 
to  meet  the  threshold  can  be  cause  for  the  concept  or  system  selection  to  be 
reevaluated  or  the  program  to  be  reassessed  or  terminated. 

The  Army  evaluates  system  performance  through  the  data  derived  from  three 
primary  types  of  tests:  contractor,  developmental  and  operational.  Contractor  tests  are 
performed  to  determine  design  and  development  maturity  in  order  to  decrease  the 
technological  risks  prior  to  initiating  formal  government  testing.  It  is  common  for 
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contractor  tests  to  be  performed  at  government  test  ranges  or  facilities  since  most 
contractors  do  not  have  facilities  adequate  to  perform  the  necessary  tests.  Performing 
these  tests  at  government  facilities  provides  a  greater  opportunity  for  government 
evaluators  to  witness  the  tests,  thereby  affording  the  evaluators  greater  confidence  in 
the  test  protocols  as  well  as  trust  in  the  authenticity  of  the  results.  In  some  cases,  this 
trust  and  confidence  in  contractor  test  data  allows  government  evaluators  to  use  this 
data  in  lieu  of  repeating  the  tests  later  during  developmental  testing. 

Developmental  tests  (DTs)  encompass  a  wide  range  of  technical  and  safety 
testing.  DT  is  intended  to  measure  the  technical  performance  of  a  system.  The  PMO 
and  the  contractor  use  DT  to  demonstrate  that  the  technical  risk  areas  identified  during 
contractor  testing  can  be  reduced  to  acceptable  levels.  With  the  exception  of  DT  safety 
tests,  developmental  tests  are  largely  discretionary  and  are  primarily  used  by  the  PMO 
and  contractor  to  identify  deficiencies  and  confirm  fixes  prior  to  more  strenuous 
operational  tests.  Specific  safety  testing  is  required  by  law  to  verify  system  safety  with  a 
high  degree  of  confidence.  Safety  documentation  must  be  provided  by  ATEC  prior  to 
any  hands  on  use,  training,  testing  or  maintenance  by  Soldiers. 

After  a  system  meets  its  technical  requirements,  Operational  Testing  (OT)  is 
used  to  assess  the  Soldier  and  system  interface  and  the  ability  of  a  Soldier  or  unit, 
equipped  with  a  system,  to  complete  their  intended  mission.  OT  provides  data  in 
support  of  answers  to  these  questions: 

•  Will  the  system  work  in  an  operational  environment? 

•  Can  the  Soldiers  use  the  system  as  it  is  intended? 

•  How  will  Soldiers  address  system  failures  or  employ  work-arounds? 
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•  Does  the  system  represent  a  net  improvement  to  operational  capability? 

•  Does  use  of  the  system  introduce  other  issues  or  problems  affecting  Mission 
capabilities?19 

Over  the  last  few  years  the  test  community  began,  where  feasible,  integrating  DT  and 
OT  into  a  combined  test  event  which  saves  time  and  money.  Integrated  testing’s  goal  is 
to  conduct  a  seamless  test  program  that  produces  credible  data  useful  to  evaluators, 
and  address  developmental,  sustainment,  and  operational  issues.  Integrated  testing 
allows  for  the  collaborative  planning  of  test  events;  where  a  single  test  point  or  mission 
can  provide  data  to  satisfy  multiple  objectives,  without  compromising  the  test  objectives 
of  participating  test  organizations. 

Acquisition  Culture 

For  decades  leaders  have  attempted  to  institutionalize  change  within  the 
acquisition  process.  Some  initiatives  have  been  popular,  widely  accepted  and  effective, 
while  others  have  had  less  effective  results.  Despite  the  fact  that  most  acquisition 
professionals  agree  that  change  is  necessary,  it  is  human  nature  to  resist  change.  As 
Secretary  Gates  indicated  “the  culture  of  any  large  organization  takes  a  long  time  to 
change.  The  really  tough  part  is  preserving  those  elements  of  the  culture  that 
strengthen  the  institution  and  motivate  the  people  in  it,  while  shedding  those  elements 
of  the  culture  that  are  barriers  to  progress  and  achieving  the  mission.”20  The  2010  Army 
Acquisition  Review  chartered  by  Secretary  McHugh  found  that  “the  Army  acquisition 
culture  has  increasingly  become  risk  averse,  placing  more  attention  on  not  repeating 
mistakes  than  on  identifying  and  managing  risk  for  the  best  outcome.”21 
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Unique  cultural  characteristics  exist  among  acquisition  professionals.  First  and 
foremost,  the  acquisition  community  generally  shares  an  overall  attitude  of  optimism. 
Optimistic  characteristics  are  evident  in  such  things  as  the  schedules  developed, 
assumptions  made,  approaches  to  working  through  problems  encountered  and 
forecasts  of  system  performance.  Whether  responding  to  inquiries  from  Congress,  the 
press  or  the  public,  most  acquisition  professionals  downplay  the  significance  of 
performance  setbacks,  cost  overruns  and  schedule  changes.22  Optimism  can  be  a 
double  edged  sword  in  this  environment.  On  one  hand,  optimistic  forecasts  and 
projections  enable  acquisition  personnel  to  more  readily  secure  financial  resources  by 
way  of  the  PPBE  process.  On  the  other  hand,  when  projections  fall  short  they  can  serve 
to  discredit  acquisition  personnel  in  the  eyes  of  Congress  and  other  senior  decision 
makers.  Additionally,  the  acquisition  culture  is  overly  bureaucratic;  a  by-product  in  large 
part  created  by  a  lack  of  trust  and  a  result  of  previous  reforms  that  created  additional 
checks,  balances  and  oversight.  Needless  to  say,  layers  of  bureaucracy  often  impose 
redundancy  and  can  lead  to  delays. 

Organizational  cultures  are  created  in  part  by  leaders,  and  one  of  the  most 
decisive  functions  of  leadership  is  the  creation,  management,  and  sometimes  even 
destruction  of  cultures.23  In  Leading  Change,  John  Kotter  indicates  that  organizational 
change  can  stall  because  of  “inwardly  focused  cultures,  paralyzing  bureaucracy,  a  low 
level  of  trust,  lack  of  teamwork,  arrogant  attitudes,  a  lack  of  leadership,  and  the  general 
fear  of  the  unknown.”24  Major  General  (MG)  Genaro  Dellarocco  overcame  those 
obstacles  and  led  ATEC  through  a  period  of  broad  organizational  change.  MG 
Dellarocco’s  arrival  and  assumption  of  command  at  ATEC  marked  a  turning  point  in  a 
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cultural  transformation  in  the  command.  Together  with  Brian  Simmons,  the  command’s 
Technical  Director,  change  was  embraced.  Testers  and  evaluators  alike  were 
encouraged  to  seek  more  efficient  practices  whether  inside  the  headquarters  or  out  on 
the  test  ranges.  The  leadership  fostered  creative  approaches  which  fundamentally 
changed  the  way  ATEC  provided  test  and  evaluation  support.  The  support  by  the 
leadership  enabled  the  test  community  in  collaboration  with  PMOs  to  develop  innovative 
solutions  that  saved  time  and  money. 

Recent  T&E  Transformation 

Throughout  the  years  there  have  been  extensive  reviews  and  studies  aimed  at 
improving  the  acquisition  system.  These  reviews  and  studies,  which  resulted  in  only 
minor  improvements,  have  not  eliminated  the  overall  critique  of  the  system  that  it  is  too 
time  consuming,  too  costly  and  unable  to  deliver  products  that  meet  the  technical 
parameters  required.  The  challenge  in  transforming  Army  acquisition  lays  primarily  in 
that  MDAPs  are  extremely  technologically  advanced  and  complex.  These  programs  are 
often  designed  to  achieve  performance  levels  never  before  realized.  The  laws  of 
physics  are  often  pushed  to  the  limit  with  the  application  and  use  of  materials  that  have 
never  before  been  used  in  military  applications,  creating  an  environment  of  risk  and 
uncertainty.  Despite  these  technical  challenges,  there  have  been  continual  attempts  to 
heed  the  call  from  Secretary  McHugh  to  transform  Army  acquisition  and  test  and 
evaluation  in  order  to  better  deliver  capabilities  to  the  Warfighter.  Many  of  these 
attempts  have  resulted  in  improvements;  however,  some  transformative  measures  have 
resulted  in  unintentional  delays  and  burdens  to  the  process.  For  example,  WSARA 
2009s  creation  of  DT&E  provided  OSD  more  oversight  of  DT  activities.25  Reform  efforts 

such  as  WSARA  instituted  more  reviewers  in  an  already  complex  acquisition  process. 
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More  reviews  add  to  the  burden  of  program  managers  and  equip  more  oversight 
agencies  with  veto  authority.  As  a  result,  rather  than  streamlining  the  acquisition 
process,  the  process  became  more  complex  and  time  consuming. 

The  acquisition  community  has  proven  that  in  times  of  war  it  can  deliver  timely 
capabilities.  This  is  primarily  due  to  the  fact  that  wars  enable  that  community  to 
streamline  the  acquisition  process,  allowing  them  to  bypass  some  of  the  lengthy 
procedural  delays  that  hinder  traditional  programs  in  peacetime.  For  instance,  to 
respond  to  the  urgent  and  rapid  demand  for  systems  to  support  the  deployment  in  Iraq 
and  Afghanistan,  ATEC  supported  rapid  acquisition  by  developing  a  new  reporting 
process  that  stressed  the  identification  and  documentation  of  the  equipment  capabilities 
and  limitations  for  decision  makers  and  the  users  prior  to  fielding  to  theater.  ATEC 
deployed  Forward  Operational  Assessment  teams,  on  six-month  rotations,  to  Iraq, 
Kuwait,  and  Afghanistan  to  collect  information  on  systems  in  order  to  identify  and  fix 
shortfalls  so  that  systems  could  quickly  fill  the  requests  from  operational  commanders.26 
Through  capabilities  and  limitation  reports  and  safety  confirmations,  ATEC  provided 
information  to  decision  makers  to  support  fielding  decisions  in  a  matter  of  months;  far 
quicker  than  it  takes  for  traditional  acquisition  programs. 

Despite  the  ability  of  rapid  acquisition  programs  to  provide  responsive 
capabilities  to  Warfighters,  there  are  inherent  problems  and  risks  associated  with  this 
process.  For  example,  rapid  acquisition  efforts  rarely  adequately  address  the  DOTMLPF 
considerations,  they  often  provide  solutions  to  short  term  problems  rather  than  long 
term  ones,  and  they  often  fail  to  account  for  the  life  cycle  sustainment  plans  and 
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associated  costs.  As  a  result,  the  Army  is  faced  with  addressing  long  term  sustainment 
costs  and  issues  to  systems  that  were  fielded  only  for  immediate  capabilities. 

ATEC  used  the  2005  Base  Realignment  and  Closure  (BRAC)  as  a  means  to 
disestablish  a  headquarters,  consolidate  staff  functions  and  reduce  overhead.  ATEC 
headquarters  moved  from  Alexandria,  VA  to  Aberdeen  Proving  Ground  (APG),  MD  as 
part  of  the  2005  BRAC.  ATEC  used  this  as  an  opportunity  to  reorganize  by  eliminating 
the  Developmental  Test  Command  (DTC)  Headquarters,  a  subordinate  command  of 
ATEC,  by  absorbing  the  functions  of  the  DTC  staff  within  the  ATEC  Headquarters  staff. 
Retirements  and  personnel  electing  not  to  BRAC  to  APG  allowed  ATEC  to  maintain 
stable  personnel  levels.  This  reorganization  was  an  example  of  how  ATEC  leaders 
identified  opportunities  to  use  resources  better,  leading  to  reduction  in  the  cost  of  doing 
business. 

In  April  2009,  a  Defense  Science  Board  report  stated  that  “fixing  the  acquisition 
process  is  a  critical  national  security  issue — requiring  the  attention  of  the  Secretary  of 
Defense.”27  General  (GEN)  Peter  Chiarelli,  Vice  Chief  of  Staff  of  the  Army  from  2008  to 
2012,  took  notice  and  challenged  Army  developmental  and  operational  testers  to 
continue  finding  ways  to  work  together  to  speed  up  acquisition  and  test  and  evaluation 
processes.  GEN  Chiarelli  brought  the  test,  PMO  and  TRADOC  communities  together  in 
201 1  by  mandating  that  the  three  organizations  use  semi-annual  network  evaluations  at 
Fort  Bliss,  TX  as  a  means  to:  integrate  components  of  the  Army’s  network,  evaluate 
these  components  and  determine  the  ability  for  these  components  to  be  fielded.28 

The  three  communities  formed  a  triad  consisting  of  ATEC,  the  Brigade 
Modernization  Command  representing  TRADOC,  and  the  Assistant  Secretary  of  the 
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Army  for  Acquisition,  Logistics  and  Technology’s  System  of  Systems  Integration 
Directorate  representing  multiple  PMOs.  They  created  a  semi-annual  evaluation  of 
technologies  referred  to  as  the  Network  Integrated  Evaluation  (NIE).  The  primary 
purpose  of  the  NIE  is  to  support  acquisition  decision  reviews  (low  rate  initial  production 
and  full  rate  production)  as  well  as  partner  with  industry  in  the  test  and  evaluation 
process.  The  NIEs  are  part  of  the  Army’s  newly  created  Agile  Process.  The  Agile 
Process  is  a  streamlined  acquisition  methodology  used  to  address  defined  capability 
gaps  and  insert  new  technologies  into  the  network.29 

The  Agile  Process  consists  of  seven  phases  (Figure  1).  The  process  starts  with 
the  identification  of  capability  gaps,  requirements  and  candidate  solutions.  This  is  a 
continuous  process.  The  process  includes  assessment  and  selection  of  solutions  to  be 
tested  and  evaluated  and  it  concludes  with  an  acquisition  decision.  Industry  is 
encouraged  to  participate  by  submitting  solutions  on  a  semi-annual  basis  that  address 
capability  gaps.  Although  this  process  is  ideally  suited  for  command,  control, 
communications,  computers  and  intelligence  (C4I)  systems  that  are  networked,  it  serves 
as  an  effective  means  to  test  the  interoperability  of  independent  systems  that  access 
the  network  in  order  to  exchange  information.  The  Agile  Process  supports  Capability 
Set  Management  which  is  a  structured  approach  that  allows  the  Army  to  buy  and  field 
equipment  that  units  need  and  then  incrementally  modernize  this  equipment  over  time. 
This  reduces  the  problem  of  expending  resources  on  technology  that  may  be  out  of  date 
by  the  time  it  is  needed.  By  employing  the  Agile  Process  with  NIE,  the  Army  leverages 
industry  advancements  and  keeps  up  with  the  pace  of  changing  technology.30  The  NIEs 
also  serve  as  a  forcing  mechanism  to  reduce  the  tendency  of  program  managers  to 
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develop  technologies  in  a  stove  piped  manner.  The  NIEs  provide  an  environment  for  the 
demonstration  of  system  of  systems  integration.  Since  most  materiel  systems  link  to  or 
rely  on  other  systems  to  accomplish  the  intended  mission,  the  NIE  encourages  materiel 
developers  to  account  for  potential  integration  and  interoperability  challenges  with  other 
fielded  systems. 
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•  Decision  Point  (DP)  1  -  Approve  Potential  Solutions  List 

•  DP 2 -Approve  Solutions  List 

•  DP  -  3  Senior  Leadership  Approves  Capability  Recommendations 


Figure  1 .  Agile  Process 


After  the  execution  of  NIE  1 2.2  in  FY201 2,  ATEC  provided  reports  for  the  35 
systems  under  evaluation.  Of  the  35,  1 5  had  high  potential  for  fielding,  1 5  had  medium 
potential  for  fielding  and  5  had  low  potential  for  fielding.31  These  reports  were  used  to 
inform  decision  makers  on  acquisition  decisions  for  these  systems.  Conducting  NIEs 
allows  ATEC  to  conduct  tests  concurrently  which  results  in  cost  savings  and  cost 
avoidance.  These  efficiencies  along  with  other  program  restructures  have  resulted  in 
over  $6  billion  in  savings.32 

Recommendations  to  Transform  T&E 

Chief  of  Staff  of  the  Army  GEN  Odierno  commented  that  the  greatest  threat  to 
our  national  security  is  fiscal  uncertainty,  but  this  uncertainty  presents  us  with  an 
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opportunity  to  shape  the  Army  of  the  future.33  There  is  opportunity  to  improve  test  and 
evaluation.  However,  this  opportunity  requires  the  collective  willingness  of  ATEC, 

PMOs,  TCMs  and  OSD  to  work  together  under  the  guidance  and  parameters  set  forth 
by  the  Secretary  of  Defense  to  improve  the  acquisition  process.  As  noted  previously, 
the  Army  test  community  has  made  positive  attempts  to  improve  the  way  it  conducts 
test  and  evaluation.  However,  more  can  be  done  with  the  cooperation  of  all 
stakeholders.  Like  ATEC,  all  organizations  involved  in  Army  acquisition  need  to  take  an 
introspective  look  and  challenge  their  processes  and  their  unique  cultural  characteristics 
that  may  inhibit  making  changes.  There  are  no  silver  bullets  that  will  fix  the  acquisition 
process,  but  the  Army  can  take  actions  to  improve  test  and  evaluation.  These  actions 
fall  into  7  categories: 

•  Allow  for  greater  access  to  and  use  of  contractor  test  data. 

•  Use  adequate  testing  to  confirm  requirements. 

•  Increased  accountability  among  stakeholders. 

•  Consolidate  reviews  and  eliminate  redundancy. 

•  Improve  the  requirements  process. 

•  Support  risk  based  testing. 

•  Include  the  T&E  community  in  configuration  steering  boards. 

Allow  for  Greater  Access  to  and  Use  of  Contractor  Data 

As  previously  mentioned,  in  some  cases  contractor  test  data  can  be  used  by 
government  evaluators,  eliminating  the  need  to  repeat  the  test  using  government 
resources.  Even  if  the  data  is  not  sufficient,  access  to  contractor  data  can  serve  to 
optimize  government  testing.  For  example,  transportability  tests  or  vulnerability  tests 
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could  be  optimized  if  government  personnel  had  access  to  the  contractor 
subcomponent  and  component  tests.  There  is  reluctance  among  contractors  to  allow 
access  because  government  evaluators  may  use  this  data  for  negative  reporting. 
However,  inserting  language  in  the  requests  for  proposal  allowing  for  government 
access  to  contractor  test  data  would  lend  to  an  increase  in  use  and  ease  of  obtainment 
of  data. 

Use  Adequate  Testing  to  Confirm  Requirements 

A  continuing  debate  among  acquisition  professionals  deals  with  the  amount  of 
rigor  that  should  be  applied  to  government  testing.  In  one  camp  proponents  of  rigor 
argue  that  research,  development,  test  and  evaluation  (RDT&E)  represents  less  than 
30%  of  a  systems  lifecycle  costs.  They  argue  that  since  70%  of  a  system’s  costs  are 
operations  and  support,  from  a  pure  economic  perspective,  it  is  better  to  identify 
deficiencies  and  allow  for  fixes  in  the  RDT&E  phase  rather  than  field  unreliable  systems 
that  require  much  more  costly  fixes  during  the  sustainment  phase.  This  line  of  thinking 
led  the  test  community  to  develop  “robust”  test  plans  that  were  effective  in  identifying 
the  full  capabilities  and  weaknesses  of  systems.  These  test  plans  were  time  consuming 
and  costly  and  drew  the  attention  of  senior  decision  makers  aimed  at  conducting  tests 
more  efficiently,  which  then  led  the  test  community  to  develop  “adequate”  test  plans. 
However,  leaders  need  to  better  define  what  adequate  means.  From  OSD’s  perspective 
“to  be  adequate,  the  operational  evaluation  must  report  performance  across  the 
operational  envelope,  not  just  at  single  conditions  specified  in  the  capabilities 
documents.”34  Many  believe  that  this  is  not  a  definition  of  adequacy,  but  rather  a 
definition  of  robust  testing  which  is  overly  costly  and  too  time  consuming. 
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The  test  community  needs  to  continue  to  verify  the  Soldier’s  ability  to  operate 
systems  in  their  intended  environment  in  accordance  with  the  requirements  documents 
in  order  to  ensure  systems  are  effective,  suitable  and  survivable.  To  the  extent  possible, 
the  test  community  should  leverage  ail  available  data  to  include  contractor,  government 
DT  and  previous  government  OT  in  cases  where  the  fit,  form  or  function  of  a  system 
has  not  been  changed  from  the  Soldiers’  perspective.  Additionally,  in  some  cases  the 
test  community  has  mandated  the  demonstration  of  all  requirements  (both  threshold 
and  objective).  In  cases  where  DOT&E  or  DT&E  mandate  that  the  Service  demonstrate 
above  threshold  level  capabilities,  OSD  should  pay  the  extra  costs.  Requiring  the 
Services  to  invest  in  systems  that  perform  above  the  threshold  capability  results  in 
mandated  requirements  from  the  test  community  to  the  Services,  which  is  contradictory 
to  the  acquisition  process. 

Increased  Accountability  Among  Stakeholders 

Requirements  and  proving  system  safety  drives  test  programs.  The  testers,  both 
Service  and  OSD,  need  to  be  accountable  for  all  T&E  actions  imposed  on  programs. 
ATEC  is  responsible  for  executing  adequate  testing  and  for  being  efficient  in  the 
process.  PMOs  are  accountable  for  all  the  costs  they  impose.  Similarly,  test 
requirements  added  by  organizations  should  be  identifiable  and  defendable.  This  is  not 
currently  the  process.  All  additions  by  OSD  to  T&E  programs  now  only  show  on  the 
Service  T&E  cost  line.  Testing  demands  without  accompanying  accountability  and 
transparency  hinder  the  advancement  of  a  T&E  cost  culture  and  fuel  the  debate  about 
how  much  testing  really  costs.35  DOT&E’s  role  is  established  in  law.  However,  there 
seems  to  be  great  latitude  in  the  interpretations  of  the  true  intent  of  Congressional 
legislation  vice  what  DOT&E  is  enacting.  A  roles  and  missions  relook  is  warranted  to 
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determine  the  cost/benefit  to  the  current  method  of  implementation  of  the  Congressional 
mandate  for  OSD  oversight  agencies. 

There  is  an  accusation  among  many  stakeholders  that  test  costs  are  too  high. 
Providing  test  budgets  to  ATEC  could  resolve  this  accusation.  Currently  RDT&E  funds 
are  managed  by  the  PMO.  Providing  RDT&E  budget  authority  to  ATEC  will  allow  the 
test  community  to  determine  how  it  manages  resources  within  constraints.  The  benefits 
to  this  process  include:  better  traceability  of  test  costs,  improved  cost  estimates  for 
future  testing  based  on  actual  test  cost  data  and  improved  obligations  and  expenditures 
for  each  program  based  on  a  better  understanding  of  test  costs. 

Consolidate  Reviews  and  Eliminate  Redundancy 

WSARA  2009  established  Developmental  Testing  (DT)  oversight  in  OSD.  This 
review  is  in  addition  to  the  existing  oversight  of  Operational  Testing  (OT)  and  Live  Fire 
efforts.  Currently,  the  DT  and  OT  oversight  agencies  are  different  offices  within  OSD, 
each  with  separate  and  distinct  chains  of  command  and  reporting  procedures.  Separate 
DT  and  OT  approvals  add  time  to  an  already  lengthy  process.  Separate  DT&E  and 
DOTE  OSD  oversight  requirements  increase  and  complicate  the  staffing  timelines  and 
briefings.  Getting  DOT&E  to  accept  DT  data,  government  and  contractor,  and  getting 
DT&E  to  wait  until  OT  for  select  data  is  challenging  and  can  lead  to  redundant  testing. 
Different  T&E  elements  in  OSD  work  in  opposition  to  integrated  T&E  planning  and 
execution.36  During  Integrated  Product  Team  meetings,  program  managers  can  have 
anywhere  from  four  to  six  test  and  evaluation  points  of  contact  to  take  guidance  from, 
two  from  OSD,  one  from  the  Deputy  Under  Secretary  of  the  Army  for  Test  and 
Evaluation  (DUSA  (TE)),  and  anywhere  from  one  to  three  within  ATEC.  This  is  a  by¬ 
product  of  years  of  acquisition  reform  that  resulted  in  more  layers  of  oversight  and 
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involvement.  OSD  should  Identify  the  similarities  in  missions  of  organizations  such  as 
DOT&E  and  DT&E,  DUSA  (TE)  and  ATEC,  and  the  engineering  test  agencies  located 
within  the  Research  and  Development  Command.  For  example,  combine  the  oversight 
responsibilities  of  DOT&E  and  DT&E  so  that  one  representative  has  oversight  of  both 
DT  and  OT  for  a  system  under  oversight  to  streamline  the  management  chain  of 
command  and  reduce  the  number  of  people  needed  to  support  program  managers.  As 
a  result,  efficiencies  can  be  gained  through  streamlined  reporting  as  well  as  reduced 
conflicting  guidance.  Too  often  program  managers  must  convene  meetings  with  DT&E 
representatives  seeking  approval  of  efforts  that  have  both  DT  and  OT  impacts  only  to 
convene  the  same  meeting  later  with  DOT&E  representatives  to  discuss  the  same 
information.  Cost  savings  and  faster  results  are  possible  by  consolidating  organizations, 
centralizing  functions  and  reducing  duplication  and  redundancy. 

Improve  the  Requirements  Process 

The  Army  needs  to  improve  the  way  it  defines  requirements  and  necessary 
capabilities.  Test  and  evaluation  starts  with  an  understanding  of  system  requirements 
and  answering  whether  a  system  meets  or  exceeds  requirements.  Unconstrained 
system  requirements,  requirement  and  technology  creep  and  a  failure  of  TRADOC 
personnel  to  understand  their  own  requirements  are  pervasive  problems.  The 
requirements  generation  process  is  primarily  handled  by  TRADOC.  System 
requirements  developed  during  World  War  II  are  often  found  within  many  of  today’s 
requirements  documents.  Rather  than  pass  operational  requirements  off  to  the  next 
generation  system,  TRADOC,  with  input  from  the  Combatant  Commands,  needs  to 
improve  the  method  with  which  it  derives  operational  requirements.  This  derivation 

needs  to  be  supported  by  data  so  that  any  shortfalls  or  tradespace  in  the  requirements 
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can  be  understood  by  decision  makers.  Requirements  need  to  be  articulated  in  terms  of 
operational  impact  so  that  in  situations  where  a  system  is  not  performing  as  expected 
the  acquisition  community  can  articulate  the  operational  impact  to  the  JROC.  The  JROC 
would  then  be  able  to  determine  how  much  the  additional  capability  is  worth  or  to  relax 
the  requirement. 

Risk-Based  Testing  and  Decision-Making 

The  entire  acquisition  community,  testers,  PMO  and  force  developers,  needs  to 
develop  an  agile  framework  that  guides  decision-making.  The  DoDI  5000.02  outlines  a 
management  system  designed  to  reduce  risk  to  the  greatest  extent  possible  so  that 
decision-makers  can  make  the  best  decisions  on  materiel  acquisitions.  However,  DoDI 
5000.02  should  not  be  the  management  system  that  guides  all  system  acquisitions. 
Many  materiel  solutions  are  upgrades  to  existing  systems  that  are  low  risk  initiatives. 
These  cases  should  be  handled  differently  than  high  risk  evolutionary  ventures.  In 
cases  of  system  upgrades  and  commercial-off-the-shelf  procurement,  the  test 
community  needs  to  more  readily  accept  previous  test  data  and  accept  modeling  and 
simulation  data,  thereby  reducing  costs  and  the  acquisition  timeline. 

Include  the  T&E  Community  in  Configuration  Steering  Boards 

Configuration  Steering  Boards  (CSB)  were  instituted  as  a  means  to  limit 
requirements  changes  and  avoid  cost  increases  to  programs.  The  attendee  list  at  CSBs 
are  intended  to  be  broad  as  the  board  reviews  the  requirements  and  configuration 
changes  that  have  the  potential  to  impact  programs.  However,  the  Service  test 
community  is  rarely  invited  to  participate.  The  agendas  of  CSBs  typically  consist  of 
requirements  overview,  cost  and  affordability,  schedule  review,  de-scoping  suggestions 
and  management  initiatives.  There  is  rarely  an  opportunity  for  the  Army’s  independent 
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test  agency  to  discuss  emerging  results  of  the  tests  performed  to  date.  Providing  the 
test  community  a  seat  at  the  CSB  would  enable  them  to  provide  input  on  how  the  de¬ 
scoping  proposals  could  impact  both  the  future  tests  and  the  evaluation  reports,  which 
would  be  beneficial  to  the  CSB  chairperson.  There  are  cases  where  de-scoping 
initiatives  did  not  achieve  their  intended  results  because  they  failed  to  take  into 
consideration  other  aspects  of  the  systems  requirements,  such  as  reliability,  which  were 
the  primary  drivers  of  the  test  and  evaluation  program.  Allowing  the  test  community  to 
participate  in  the  full  CSB  process  will  provide  decision-makers  with  greater  information 
and  insight. 

Conclusion 

The  Army  must  continue  to  challenge  the  processes  and  procedures  it  uses  to 
acquire  materiel  solutions.  Reliance  on  old  practices  employed  in  periods  of  economic 
prosperity  is  unaffordable.  However,  successfully  changing  processes  and  procedures 
requires  buy-in  and  synchronization  across  multiple  echelons  of  command  and  across 
diverse  management  structures.  Although  each  share  a  common  endstate,  the 
participants  in  the  Defense  Acquisition  System  have  unique  responsibilities,  varying 
interpretation  of  risk,  differing  time  horizons,  and  competing  priorities.  The  system  also 
requires  strong  leaders  who  are  willing  to  challenge  the  status  quo,  make  tough 
decisions  and  be  accountable  for  their  actions.  Senior  civilians  and  officers  who  develop 
the  momentum  to  affect  change  are  often  followed  by  those  who  are  influenced  by 
differing  priorities. 

Program  managers,  TRADOC  capability  mangers  and  T&E  personnel  have 

worked  through  challenges  in  order  to  deliver  systems  that  provide  the  Warfighter  the 

technology  edge  on  the  battlefield.  These  three  communities  must  now  work  together  to 
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improve  their  ability  to  deliver  effective,  suitable  and  survivable  systems  in  a  timely  and 
affordable  manner. 
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